Parse triple quoted string annotations as if parenthesized #15387

Glyphack · 2025-01-09T23:37:26Z

Summary

Resolves #9467

Parse quoted annotations as if the string content is inside parenthesis.
With this logic x and y in this example are equal:

y: """
   int |
   str
"""

z: """(
    int |
    str
)
"""

Also this rule only applies to triple quotes(link).

This PR is based on the comments on the issue.

I did one extra change, since we don't want any indentation tokens I am setting the State::Other as the initial state of the Lexer.

Remaining work:

Add a test case for red-knot.
Add more tests.

Test Plan

Added a test which previously failed because quoted annotation contained indentation.
Added an mdtest for red-knot.
Updated previous test.

github-actions · 2025-01-09T23:52:33Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

warning: Detected debug build without --no-cache.
error: Failed to read examples/Assistants_API_overview_python.ipynb: Expected a Jupyter Notebook, which must be internally stored as JSON, but this file isn't valid JSON: expected `,` or `]` at line 197 column 8

Formatter (preview)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

ruff format --preview

warning: Detected debug build without --no-cache.
error: Failed to read examples/Assistants_API_overview_python.ipynb: Expected a Jupyter Notebook, which must be internally stored as JSON, but this file isn't valid JSON: expected `,` or `]` at line 197 column 8

crates/ruff_python_parser/src/lib.rs

carljm

Thank you!! The red-knot test and changes look fine to me, but that's a small part of this PR. Would rather have @dhruvmanila take a look at the parser/ruff side.

crates/red_knot_python_semantic/resources/mdtest/annotations/string.md

MichaReiser

Thank you. Would you mind adding a test where an expression is incorrectly parenthesized?

def f(
	a: """
		int | 
str)
"""
)

or

def f(
	a: """
		int) | 
str
"""
)

I'm curious how it recovers in those cases and if we need to do something more than just setting nesting = 1

crates/ruff_python_parser/src/lib.rs

Glyphack · 2025-01-14T18:45:34Z

@MichaReiser I added the test cases you suggested and the tests pass(see this commit)
But I think that is because the parser is recovering.

So I fixed the nesting check to check for nesting value of 1 when mode is ParenthesizedExpression. This way the lexer will detect the problem that nesting is one level lower than started as well.

crates/ruff_python_parser/src/lexer.rs

crates/ruff_python_parser/src/lib.rs

dhruvmanila · 2025-01-15T05:33:04Z

crates/ruff_linter/resources/test/fixtures/pyflakes/F722.py

+# single quotes are not implicitly parenthesized
+invalid: "\n int"


Can we add the same test case using triple-quoted string which will be useful as a documentation?

# but, using the same annotation inside a triple-quoted string is valid valid: """\n int"""

Actually I made a mistake here, invalid: "\n int" is in a python file. So this string will not actually have a newline character but \ and n as characters. So also """"\n int""" fails.
Is there a way to actually create a type annotation that contains newline inside a python file without using triple quotes?

https://pyright-play.net/?code=JYOwbghgNsAmBcACARKgOiRoAurkCh9RIYEVUiRcKg

I removed this right now. Since I could not figure out how to do it correctly.

I see, that makes sense. I don't think it's even possible to have a newline in a single-quoted string without the \n character. This will also utilize parse_complex_annotation and not parse_simple_annotation.

crates/ruff_linter/resources/test/fixtures/pyflakes/F722.py

crates/ruff_python_parser/src/lexer.rs

MichaReiser

Thanks. This looks great.

crates/ruff_python_parser/src/lexer.rs

Co-authored-by: Dhruv Manilawala <[email protected]>

Co-authored-by: Micha Reiser <[email protected]>

dhruvmanila · 2025-01-16T06:06:24Z

...f_linter/src/rules/pyflakes/snapshots/ruff_linter__rules__pyflakes__tests__F722_F722.py.snap

+F722.py:30:11: F722 Syntax error in forward annotation: `
+		int |
+str)
+`


Unrelated to this PR, I think it would be useful to provide the syntax error that was raised during parsing this string content.

dhruvmanila

Thank you for working on this!

## Summary Ref: #15387 (comment) This PR updates `F722` to show syntax error message instead of the string content. I think it's more useful to show the syntax error message than the string content. In the future, when the diagnostics renderer is more capable, we could even highlight the exact location of the syntax error along with the annotation string. This is also in line with how we show the diagnostic in red knot. ## Test Plan Update existing test snapshots.

Glyphack force-pushed the stringized-annotations branch from b33cf05 to efbd25e Compare January 9, 2025 23:43

Glyphack force-pushed the stringized-annotations branch 2 times, most recently from 88460a5 to be3115f Compare January 13, 2025 22:12

Glyphack commented Jan 13, 2025

View reviewed changes

crates/ruff_python_parser/src/lib.rs Outdated Show resolved Hide resolved

Glyphack marked this pull request as ready for review January 13, 2025 22:39

Glyphack requested review from carljm, MichaReiser, AlexWaygood, sharkdp and dhruvmanila as code owners January 13, 2025 22:39

Glyphack changed the title ~~Parse Quoted annotations as if parenthesized~~ Parse triple quoted annotations as if parenthesized Jan 13, 2025

Glyphack changed the title ~~Parse triple quoted annotations as if parenthesized~~ Parse triple quoted string annotations as if parenthesized Jan 13, 2025

carljm reviewed Jan 13, 2025

View reviewed changes

crates/red_knot_python_semantic/resources/mdtest/annotations/string.md Outdated Show resolved Hide resolved

MichaReiser reviewed Jan 14, 2025

View reviewed changes

crates/ruff_python_parser/src/lib.rs Outdated Show resolved Hide resolved

MichaReiser added the parser Related to the parser label Jan 14, 2025

Glyphack force-pushed the stringized-annotations branch from 93125a2 to b3cf440 Compare January 14, 2025 18:21

dhruvmanila reviewed Jan 15, 2025

View reviewed changes

crates/ruff_python_parser/src/lexer.rs Outdated Show resolved Hide resolved

MichaReiser approved these changes Jan 15, 2025

View reviewed changes

crates/ruff_python_parser/src/lexer.rs Outdated Show resolved Hide resolved

Glyphack requested a review from dhruvmanila January 15, 2025 20:44

Glyphack added 8 commits January 15, 2025 21:54

Parse Quoted annotations as if parenthesized

ce05eef

Use parse_parenthesized_expression_range in red-knot

1117ca5

Do not implicitly parenthesize single quote annotations

af5a1ce

Create parse_as_string_annotation method

334ca52

Update example

8c5a6d1

Clippy fix

25c1535

Fix doctest

4b7cb1d

Add test cases for invalid annotations

96c79ff

Glyphack and others added 10 commits January 15, 2025 21:54

Document ParenthesizedExpression

63fc9ce

Check for nesting level 1 when mode is ParenthesizedExpression

d513d73

Reformat code

b6a7155

Update crates/ruff_python_parser/src/lib.rs

5b69632

Co-authored-by: Dhruv Manilawala <[email protected]>

Update crates/ruff_python_parser/src/lexer.rs

b0de204

Co-authored-by: Dhruv Manilawala <[email protected]>

Update crates/ruff_python_parser/src/lexer.rs

d023a5b

Co-authored-by: Micha Reiser <[email protected]>

Add tests and documentation

e93f59f

rename to parse_as_string_annotation

982eb2a

Update rust doc

f9c2401

Fix clippy

a74d41e

Glyphack force-pushed the stringized-annotations branch from cf342b5 to a74d41e Compare January 15, 2025 20:55

Fix test

964cafa

dhruvmanila reviewed Jan 16, 2025

View reviewed changes

dhruvmanila approved these changes Jan 16, 2025

View reviewed changes

dhruvmanila merged commit cf4ab7c into astral-sh:main Jan 16, 2025
21 checks passed

dhruvmanila mentioned this pull request Jan 16, 2025

[pyflakes] Show syntax error message for F722 #15523

Merged

BrewTestBot mentioned this pull request Jan 16, 2025

ruff 0.9.2 Homebrew/homebrew-core#204488

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parse triple quoted string annotations as if parenthesized #15387

Parse triple quoted string annotations as if parenthesized #15387

Glyphack commented Jan 9, 2025 •

edited

Loading

github-actions bot commented Jan 9, 2025 •

edited

Loading

carljm left a comment

MichaReiser left a comment

Glyphack commented Jan 14, 2025

dhruvmanila Jan 15, 2025

Glyphack Jan 15, 2025 •

edited

Loading

Glyphack Jan 15, 2025

dhruvmanila Jan 16, 2025

MichaReiser left a comment

dhruvmanila Jan 16, 2025

dhruvmanila left a comment

		# single quotes are not implicitly parenthesized
		invalid: "\n int"

Parse triple quoted string annotations as if parenthesized #15387

Parse triple quoted string annotations as if parenthesized #15387

Conversation

Glyphack commented Jan 9, 2025 • edited Loading

Summary

Test Plan

github-actions bot commented Jan 9, 2025 • edited Loading

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

carljm left a comment

Choose a reason for hiding this comment

MichaReiser left a comment

Choose a reason for hiding this comment

Glyphack commented Jan 14, 2025

dhruvmanila Jan 15, 2025

Choose a reason for hiding this comment

Glyphack Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Glyphack Jan 15, 2025

Choose a reason for hiding this comment

dhruvmanila Jan 16, 2025

Choose a reason for hiding this comment

MichaReiser left a comment

Choose a reason for hiding this comment

dhruvmanila Jan 16, 2025

Choose a reason for hiding this comment

dhruvmanila left a comment

Choose a reason for hiding this comment

Glyphack commented Jan 9, 2025 •

edited

Loading

github-actions bot commented Jan 9, 2025 •

edited

Loading

`ruff-ecosystem` results

Glyphack Jan 15, 2025 •

edited

Loading